Systems and methods for template-based generation of personalized videos

ABSTRACT

Disclosed are systems and methods for template-based generation of personalized videos. An example method may commence with receiving video configuration data including a sequence of frame images, a sequence of face area parameters defining positions of a face area in the frame images, and a sequence of facial landmark parameters defining positions of facial landmarks in the frame images. The method may continue with receiving an image of a source face. The method may further include generating an output video. The generation of the output video may include modifying a frame image of the sequence of frame images. Specifically, the image of the source face may be modified to obtain a further image featuring the source face adopting a facial expression corresponding to the facial landmark parameters. The further image may be inserted into the frame image at a position determined by face area parameters corresponding to the frame image.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation-in-part of U.S. application Ser. No.16/594,771, entitled “Systems and Methods for Providing PersonalizedVideos”, filed on Oct. 7, 2019, which is a Continuation-in-part of U.S.patent application Ser. No. 16/251,436, entitled “Systems and Methodsfor Face Reenactment,” filed on Jan. 18, 2019, a U.S. application Ser.No. 16/594,690, entitled “Systems and Methods for Providing PersonalizedVideos Featuring Multiple Persons”, filed on Oct. 7, 2019, which is aContinuation-in-part of U.S. patent application Ser. No. 16/251,436,entitled “Systems and Methods for Face Reenactment,” filed on Jan. 18,2019, a Continuation-in-Part of U.S. patent application Ser. No.16/251,472, entitled “Systems and Methods for Photorealistic Real-TimePortrait Animation,” filed on Jan. 18, 2019, and U.S. patent applicationSer. No. 16/551,756, entitled “Entertaining Mobile Application forAnimating a Single Image of a Human Body and Applying Effects,” filed onAug. 27, 2019, which is a Continuation-in-part of U.S. patentapplication Ser. No. 16/434,185, entitled “Single Image-Based Real-TimeBody Animation”, filed on Jun. 7, 2019. The present application isrelated to U.S. application Ser. No. ______, titled “System and Methodsfor Generating Personalized Videos with Customized Text Messages”, filedon ______. The aforementioned applications are incorporated herein byreference in their entirety for all purposes.

TECHNICAL FIELD

This disclosure generally relates to digital image processing. Moreparticularly, this disclosure relates to methods and systems fortemplate-based generation of personalized videos.

BACKGROUND

Sharing media, such as stickers and emojis, has become a standard optionin messaging applications (also referred herein to as messengers).Currently, some of the messengers provide users with an option forgenerating and sending images and short videos to other users via acommunication chat. Certain existing messengers allow users to modifythe short videos prior to transmission. However, the modifications ofthe short videos provided by the existing messengers are limited tovisualization effects, filters, and texts. The users of the currentmessengers cannot perform complex editing, such as, for example,replacing one face with another face. Such editing of the videos is notprovided by current messengers and requires sophisticated third-partyvideo editing software.

SUMMARY

This section is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription section. This summary is not intended to identify keyfeatures or essential features of the claimed subject matter, nor is itintended to be used as an aid in determining the scope of the claimedsubject matter.

According to one embodiment of the disclosure, a system fortemplate-based generation of personalized videos is disclosed. Thesystem may include at least one processor and a memory storingprocessor-executable codes. The at least one processor may be configuredto receive, by a computing device, video configuration data. The videoconfiguration data may include a sequence of frame images, a sequence offace area parameters defining positions of a face area in the frameimages, and a sequence of facial landmark parameters defining positionsof facial landmarks in the frame images. Each of the facial landmarkparameters may correspond to a facial expression. The at least oneprocessor may be configured to receive, by the computer device, an imageof a source face. The at least one processor may be configured togenerate, by the computing device, an output video. The generation ofthe output video may include modifying a frame image of the sequence offrame images. Specifically, the image of the source face may be modifiedbased on facial landmark parameters corresponding to the frame image toobtain a further image featuring the source face adopting a facialexpression corresponding to the facial landmark parameters. The furtherimage may be inserted into the frame image at a position determined byface area parameters corresponding to the frame image.

According to one example embodiment, a method for template-basedgeneration of personalized videos is disclosed. The method may commencewith receiving, by a computing device, video configuration data. Thevideo configuration data may include a sequence of frame images, asequence of face area parameters defining positions of a face area inthe frame images, and a sequence of facial landmark parameters definingpositions of facial landmarks in the frame images. Each of the faciallandmark parameters may correspond to a facial expression. The methodmay continue with receiving, by the computer device, an image of asource face. The method may further include generating, by the computingdevice, an output video. The generation of the output video may includemodifying a frame image of the sequence of frame images. Specifically,the image of the source face may be modified to obtain a further imagefeaturing the source face adopting a facial expression corresponding tothe facial landmark parameters. The modification of the image may beperformed based on facial landmark parameters corresponding to the frameimage. The further image may be inserted into the frame image at aposition determined by face area parameters corresponding to the frameimage.

According to yet another aspect of the disclosure, there is provided anon-transitory processor-readable medium, which storesprocessor-readable instructions. When the processor-readableinstructions are executed by a processor, they cause the processor toimplement the above-mentioned method for template-based generation ofpersonalized videos.

Additional objects, advantages, and novel features of the examples willbe set forth in part in the description which follows, and in part willbecome apparent to those skilled in the art upon examination of thefollowing description and the accompanying drawings or may be learned byproduction or operation of the examples. The objects and advantages ofthe concepts may be realized and attained by means of the methodologies,instrumentalities and combinations particularly pointed out in theappended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example and not limitation in thefigures of the accompanying drawings, in which like references indicatesimilar elements.

FIG. 1 is a block diagram showing an example environment wherein systemsand methods for template-based generation of personalized videos can beimplemented.

FIG. 2 is a block diagram showing an example embodiment of a computingdevice for implementing methods for template-based generation ofpersonalized videos.

FIG. 3 is a flow chart showing a process for template-based generationof personalized videos, according to some example embodiments of thedisclosure.

FIG. 4 is a flow chart showing functionality of a system fortemplate-based generation of the personalized videos, according to someexample embodiments of the disclosure.

FIG. 5 is a flow chart showing a process of generation of live actionvideos for use in the generation of video templates, according to someexample embodiments.

FIG. 6 shows frames of example live action videos for generating videotemplates, according to some example embodiments.

FIG. 7 shows an original image of a face and an image of the face withnormalized illumination, according to an example embodiment.

FIG. 8 shows a segmented head image, the head image with faciallandmarks, and a facial mask, according to an example embodiment.

FIG. 9 shows a frame featuring a user face, a skin mask, and a result ofrecoloring the skin mask, according to an example embodiment.

FIG. 10 shows an image of a facial image of a face synchronizationactor, an image of the face synchronization actor's facial landmarks, animage of a user's facial landmarks, and image of the user's face withthe facial expression of the face synchronization actor, according to anexample embodiment.

FIG. 11 shows a segmented face image, a hair mask, a hair mask warped toa target image, and the hair mask applied to the target image, accordingto an example embodiment.

FIG. 12 shows an original image of an eye, an image with reconstructedsclera of the eye, an image with reconstructed iris, and an image withreconstructed moved iris, according to an example embodiment.

FIGS. 13-14 show frames of example personalized video generated based onvideo templates, according to some example embodiments.

FIG. 15 is a flow chart showing a method for template-based generationof personalized videos, in accordance with an example embodiment.

FIG. 16 shows an example computer system that can be used to implementmethods for template-based generation of personalized videos.

DETAILED DESCRIPTION

The following detailed description of embodiments includes references tothe accompanying drawings, which form a part of the detaileddescription. Approaches described in this section are not prior art tothe claims and are not admitted prior art by inclusion in this section.The drawings show illustrations in accordance with example embodiments.These example embodiments, which are also referred to herein as“examples,” are described in enough detail to enable those skilled inthe art to practice the present subject matter. The embodiments can becombined, other embodiments can be utilized, or structural, logical andoperational changes can be made without departing from the scope of whatis claimed. The following detailed description is, therefore, not to betaken in a limiting sense, and the scope is defined by the appendedclaims and their equivalents.

For purposes of this patent document, the terms “or” and “and” shallmean “and/or” unless stated otherwise or clearly intended otherwise bythe context of their use. The term “a” shall mean “one or more” unlessstated otherwise or where the use of “one or more” is clearlyinappropriate. The terms “comprise,” “comprising,” “include,” and“including” are interchangeable and not intended to be limiting. Forexample, the term “including” shall be interpreted to mean “including,but not limited to.”

This disclosure relates to methods and systems for template-basedgeneration of personalized videos. The embodiments provided in thisdisclosure solve at least some issues of known art. The presentdisclosure can be designed to work on mobile devices, such assmartphones, tablet computers, or mobile phones, in real-time, althoughthe embodiments can be extended to approaches involving a web service ora cloud-based resource. Methods described herein can be implemented bysoftware running on a computer system and/or by hardware utilizingeither a combination of microprocessors or other specifically designedapplication-specific integrated circuits (ASICs), programmable logicdevices, or any combinations thereof. In particular, the methodsdescribed herein can be implemented by a series of computer-executableinstructions residing on a non-transitory storage medium such as a diskdrive or computer-readable medium.

Some embodiments of the disclosure may allow generating personalizedvideos in real time on a user computing device, such as a smartphone. Apersonalized video may be generated in the form of an audiovisual media(e.g., a video, an animation, or any other type of media) that featuresa face of a user or faces of multiple users. The personalized videos canbe generated based on pre-generated video templates. A video templatemay include video configuration data. The video configuration data mayinclude a sequence of frame images, a sequence of face area parametersdefining positions of a face area in the frame images, and a sequence offacial landmark parameters defining positions of facial landmarks in theframe images. Each of the facial landmark parameters may corresponds toa facial expression. The frame images can be generated based on ananimation video or a live action video. The facial landmark parameterscan be generated based on another live action video featuring a face ofan actor (also called a face synchronization (facesync) as described inmore detail below), animation video, audio file, text, or manually.

The video configuration file may also include sequence of skin masks.The skin masks may define a skin area of a body of an actor featured inthe frame images or a skin area of 2D/3D animation of a body. In anexample embodiment, the skin mask and the facial landmark parameters canbe generated based on two different live action videos capturingdifferent actors (referred to herein as an actor and facesync actor,respectively). The video configuration data may further include asequence of mouth region images and a sequence of eye parameters. Theeye parameters may define positions of an iris in a sclera of a facesyncactor featured in the frame images. The video configuration data mayinclude a sequence of head parameters defining a rotation and a turn ofa head, a position, a scale, and other parameters of the head. A usermay keep his head still when taking an image and look directly at thecamera, therefore, the scale and rotations of the head may be adjustedmanually. The head parameters can be transferred from a different actor(also referred to herein as a facesync actor). As used herein, afacesync actor is a person whose facial landmark parameters are beingused, an actor is another person whose body is being used in a videotemplate and whose skin may be recolored, and a user is a person whotakes an image of his/her face to generate a personalized video. Thus,in some embodiments, the personalized video includes the face of theuser modified to have facial expressions of the facesync actor andincludes the body of the actor taken from the video template andrecolored to match the color of the face of the user. The videoconfiguration data include a sequence of animated object images.Optionally, the video configuration data includes a soundtrack and/orvoice.

The pre-generated video templates can be stored remotely in acloud-based computing resource and can be downloadable by a user of acomputing device (such as a smartphone). The user of the computing maycapture, by the computing device, an image of a face or select an imageof the face from a camera roll, from a prepared collection of images, orvia the web link. In some embodiments, the image may include an animalinstead of a face of a person or may be in the form of a drawn picture.Based on the image of the face and one of the pre-generated videotemplates, the computing device may further generate a personalizedvideo. The user may send the personalized video, via a communicationchat, to another user of another computing device, share on socialmedia, download to a local storage of a computing device, or upload to acloud storage or a video sharing service.

According to one embodiment of the disclosure, an example method fortemplate-based generation of personalized videos may include receiving,by a computing device, video configuration data. The video configurationdata may include a sequence of frame images, a sequence of face areaparameters defining positions of a face area in the frame images, and asequence of facial landmark parameters defining positions of faciallandmarks in the frame images. Each of the facial landmark parametersmay correspond to a facial expression of a facesync actor. The methodmay continue with receiving an image of a source face and generating, bythe computing device, an output video. The generation of the outputvideo may include modifying a frame image of the sequence of frameimages. The modification of the frame image may include modifying theimage of the source face to obtain a further image featuring the sourceface adopting a facial expression corresponding to the facial landmarkparameters and inserting the further image into the frame image at aposition determined by face area parameters corresponding to the frameimage. Additionally, the source face may be modified, e.g., by changingcolor, making eyes bigger, and so forth. The image of the source facemay be modified based on facial landmark parameters corresponding to theframe image.

Referring now to the drawings, example embodiments are described. Thedrawings are schematic illustrations of idealized example embodiments.Thus, the example embodiments discussed herein should not be understoodas limited to the particular illustrations presented herein; rather,these example embodiments can include deviations and differ from theillustrations presented herein as shall be evident to those skilled inthe art.

FIG. 1 shows an example environment 100, in which a system and a methodfor template-based generation of personalized videos can be implemented.The environment 100 may include a computing device 105, a user 102, acomputing device 110, a user 104, a network 120, and a messengerservices system 130. The computing device 105 and computing device 110can refer to a mobile device such as a mobile phone, smartphone, ortablet computer. In further embodiments, the computing device 110 canrefer to a personal computer, laptop computer, netbook, set top box,television device, multimedia device, personal digital assistant, gameconsole, entertainment system, infotainment system, vehicle computer, orany other computing device.

The computing device 105 and the computer device 110 can becommunicatively connected to messenger services system 130 via thenetwork 120. The messenger services system 130 can be implemented as acloud-based computing resource(s). The messenger services system 130 caninclude computing resource(s) (hardware and software) available at aremote location and accessible over a network (e.g., the Internet). Thecloud-based computing resource(s) can be shared by multiple users andcan be dynamically re-allocated based on demand. The cloud-basedcomputing resources can include one or more server farms/clustersincluding a collection of computer servers which can be co-located withnetwork switches and/or routers.

The network 120 may include any wired, wireless, or optical networksincluding, for example, the Internet, intranet, local area network(LAN), Personal Area Network (PAN), Wide Area Network (WAN), VirtualPrivate Network (VPN), cellular phone networks (e.g., Global System forMobile (GSM) communications network, and so forth.

In some embodiments of the disclosure, the computing device 105 can beconfigured to enable a communication chat between the user 102 and theuser 104 of the computing device 110. During the communication chat, theuser 102 and the user 104 may exchange text messages and videos. Thevideos may include personalized videos. The personalized videos can begenerated based on pre-generated video templates stored in the computingdevice 105 or the computing device 110. In some embodiments, thepre-generated video templates can be stored in the messenger servicessystem 130 and downloaded to the computing device 105 or the computingdevice 110 on demand.

The messenger services system 130 may include a system 140 forpre-processing videos. The system 140 may generate video templates basedon animation videos or live action videos. The messenger services system130 may include a video templates database 145 for storing the videotemplates. The video templates can be downloaded to the computing device105 or the computing device 110.

The messenger services system 130 may be also configured to store userprofiles 135. The user profiles 135 may include images of the face ofthe user 102, images of the face of the user 104, and images of faces ofother persons. The images of the faces can be downloaded to thecomputing device 105 or the computing device 110 on demand and based onpermissions. Additionally, the images of the face of the user 102 can begenerated using the computing device 105 and stored in a local memory ofthe computing device 105. The images of the faces can be generated basedon other images stored in the computing device 105. The images of thefaces can be further used by the computing device 105 to generatepersonalized videos based on the pre-generated video templates.Similarly, the computing device 110 may be used to generate images ofthe face of the user 104. The images of the face of the user 104 can beused to generate personalized videos on the computing device 110. Infurther embodiments, the images of the face of user 102 and images ofthe face of the user 104 can be mutually used to generate personalizedvideos on the computing device 105 or the computing device 110.

FIG. 2 is a block diagram showing an example embodiment of a computingdevice 105 (or computing device 110) for implementing methods forpersonalized videos. In the example shown in FIG. 2, the computingdevice 110 includes both hardware components and software components.Particularly, the computing device 110 includes a camera 205 or anyother image-capturing device or scanner to acquire digital images. Thecomputing device 110 can further include a processor module 210 and astorage module 215 for storing software components andprocessor-readable (machine-readable) instructions or codes, which whenperformed by the processor module 210 cause the computing device 105 toperform at least some steps of methods for template-based generation ofpersonalized videos as described herein. The computing device 105 mayinclude a graphical display system 230 and a communication module 240.In other embodiments, the computing device 105 may include additional ordifferent components. Moreover, the computing device 105 can includefewer components that perform functions similar or equivalent to thosedepicted in FIG. 2.

The computing device 110 can further include a messenger 220 forenabling communication chats with another computing device (such as thecomputing device 110) and a system 250 for template-based generation ofpersonalized videos. The system 250 is described in more detail belowwith reference to FIG. 4. The messenger 220 and the system 250 may beimplemented as software components and processor-readable(machine-readable) instructions or codes stored in the memory storage215, which when performed by the processor module 210 cause thecomputing device 105 to perform at least some steps of methods forproviding communication chats and generation of personalized videos asdescribed herein.

In some embodiments, the system 250 for template-based generation ofpersonalized videos can be integrated in the messenger 220. A userinterface of the messenger 220 and the system 250 for template-basedpersonalized videos can be provided via the graphical display system230. The communication chats can be enabled via the communication module240 and the network 120. The communication module 240 may include a GSMmodule, a WiFi module, a Bluetooth™ module, and so forth.

FIG. 3 is a flow chart showing steps of a process 300 for template-basedgeneration of personalized videos, according to some example embodimentof the disclosure. The process 300 may include production 305,post-production 310, resources preparation 315, skin recoloring 320, lipsynchronization and facial reenactment 325, hair animation 330, eyesanimation 335, and deploy 340. The resource preparation 315 can beperformed by the system 140 for pre-processing videos in the messengerservices system 130 (shown in FIG. 1). The resource preparation 315results in generating video templates that may include videoconfiguration data.

The skin recoloring 320, lip synchronization and facial reenactment 325,hair animation 330, eyes animation 335, and deploy 340 can be performedby the system 250 for template-based generation of personalized videosin computing device 105 (shown in FIG. 2). The system 250 may receive animage of the user's face and video configuration data and generate apersonalized video featuring the user's face.

The skin recoloring 320, lip synchronization and facial reenactment 325,hair animation 330, eyes animation 335, and deploy 340 can be alsoperformed by the system 140 for pre-processing videos in messengerservices system 130. The system 140 can receive test images of userfaces and a video configuration file. The system 140 may generate testpersonalized videos featuring the user faces. The test personalizedvideos can be reviewed by an operator. Based on a result of the review,the video configuration file can be stored in the video templatesdatabase 145 and can then be downloaded to the computing device 105 orcomputing device 110.

The production 305 may include idea and scenario creation,pre-production during which a location, props, actors, costumes andeffects are identified, and production itself, which can require one ormore recording sessions. In some example embodiments, the recording maybe performed by recording a scene/actor on a chroma key background, alsoreferred herein to as a green screen or chroma key screen. To allow thesubsequent head tracking and resources clean-up, the actors may wearchroma key face masks (e.g., balaclavas) with tracking marks that coverthe face of the actors, but leave the neck and the bottom of the chinopen. The idea and scenario creation are shown in detail in FIG. 5.

In an example embodiment, the steps of pre-production and subsequentproduction 305 is optional. Instead of recording an actor, 2D or 3Danimation may be created or third-party footages/images may be used.Furthermore, an original background of the image of the user may beused.

FIG. 5 is a block diagram showing a process 500 of generating liveaction videos. The live action videos can be further used to generatevideo templates for generation of personalized video. The process 500may include generating an idea at step 505 and creating a scenario atstep 510. The process 500 may continue with pre-production at step 515,which is followed by production 305. The production 305 may includerecording using a chroma key screen 525 or at a real life location 530.

FIG. 6 shows frames of example live action videos for generating videotemplates. Frames for video 605 and video 615 are recorded at a reallife location 530. Frames for video 610, video 620, and video 625 arerecorded using a chroma key screen 525. The actors may wear chroma keyface masks 630 with tracking marks that cover the face of the actors.

The post-production 310 may include video editing or animation, visualeffects, clean-up, sound design and voice over recording.

During the resources preparation 315, the resources prepared for furtherdeploy may include the following components: a background footagewithout a head of an actor (i.e., preparing a cleaned-up backgroundwhere the head of the actor is removed); a footage with an actor on ablack background (only for recorded personalized videos); a foregroundsequence of frames; an example footage with a generic head andsoundtrack; coordinates for head position, rotation, and scale; animatedelements that are attached to the head (optional); soundtracks with andwithout a voice-over; a voice-over in a separate file (optional); and soforth. All of these components are optional and may be rendered indifferent formats. The number and configuration of the components dependon the format of the personalized video. For example, a voice-over isnot needed for customized personalized videos, background footages andhead coordinates are not needed if the original background from apicture of the user is used, and so forth. In an example embodiment, thearea where the face needs to be located may be indicated (e.g.,manually) instead of preparing a file with coordinates.

The skin recoloring 320 allows to match the color of a skin of the actorin the personalized video to the color of a face on an image of theuser. To implement this step, skin masks that indicate specificallywhich part of background has to be recolored may be prepared. It may bepreferable to have a separate mask for each body part of the actor(neck, left and right hands, etc.).

The skin recoloring 320 may include facial image illuminationnormalization. FIG. 7 shows an original image 705 of a face and an image710 of the face with normalized illumination, according to an exampleembodiment. Shadows or highlights caused by uneven illumination affectcolor distribution and may lead to too dark or too light of a skin toneafter recoloring. To avoid this, shadows and highlights in the face ofthe user may be detected and removed. The facial image illuminationnormalization process includes the following steps. An image of a faceof the user may be transformed using a deep convolutional neuralnetwork. The network may receive an original image 705 in the form of aportrait image taken under arbitrary illumination and change theillumination of the original image 705 to make the original image 705evenly illuminated while keeping the subject in original image 705 thesame. Thus, the input of the facial image illumination normalizationprocess includes the original image 705 in the form of the image of theface of the user and facial landmarks. The output of the facial imageillumination normalization process includes the image 710 of the facewith normalized illumination.

The skin recoloring 320 may include mask creation and body statistics.There may only be a mask for the whole skin or separate masks for bodyparts. Also, different masks can be created for different scenes in thevideo (e.g., due to significant illumination change). Masks may becreated semi-automatically, e.g., by such technologies as keying, withsome human guidance. Prepared masks may be merged into video assets andthen used in the recoloring. Also, to avoid unnecessary computations inreal-time, color statistics may be calculated for each mask in advance.Statistics may include mean value, median value, standard deviation, andsome percentiles for each color channel. Statistics can be computed inRed, Green, Blue (RGB) color space as well as in the other color spaces(Hue, Saturation, Value (HSV) color space, CIELAB color space (alsoknown as CIE L*a*b* or abbreviated as “LAB” color space), etc.). Theinput of the mask creation process may include grayscale masks for bodyparts of an actor with uncovered skin in the form of videos or imagesequences. The output of the mask creation process may include maskscompressed and merged to videos and color statistics per each mask.

The skin recoloring 320 may further include facial statisticscomputation. FIG. 8 shows a segmented head image 805, the segmented headimage 805 with facial landmarks 810, and a facial mask 815, according toan example embodiment. Based on segmentation of the head image of theuser and facial landmarks, the facial mask 815 of the user may becreated. Regions such as eyes, mouth, hair, or accessories (likeglasses) may be not included in the facial mask 815. The segmented headimage 805 of the user and the facial mask may be used to compute thestatistics for facial skin of the user. Thus, the input of the facialstatistics computation may include the segmented head image 805 of theuser, facial landmarks 810, and facial segmentation, and the output ofthe facial statistics computation may include color statistics for thefacial skin of the user.

The skin recoloring 320 may further include skin-tone matching andrecoloring. FIG. 9 shows a frame 905 featuring a user face, a skin mask910, a result 915 of recoloring the skin mask 910, according to anexample embodiment. The skin-tone matching and recoloring may beperformed using statistics that describe color distributions in theactor's skin and user's skin, and recoloring of a background frame maybe performed in real-time on a computing device. For each color channel,distribution matching may be performed and values of background pixelsmay be modified in order to make the distribution of transformed valuesclose to the distribution of facial values. Distribution matching may beperformed either under assumption that color distribution is normal orby applying techniques like multidimensional probability densityfunction transfer. Thus, the input of the skin-tone matching andrecoloring process may include a background frame, actor skin masks forthe frame, actor body skin color statistics for each mask, and userfacial skin color statistics, and the output may include the backgroundframe with all body parts with uncovered skin recolored.

In some embodiments, to apply skin recoloring 320, several actors withdifferent skin tones may be recorded and then a version of thepersonalized video that has the closest skin tone to the skin tone ofthe image of the user may be used.

In an example embodiment, instead of skin recoloring 320, apredetermined lookup table (LUT) may be used to adjust the color of theface to the illumination of a scene. The LUT may be also used to changethe color of the face, for example, to make the face green.

The lip synchronization and facial reenactment 325 may result inphotorealistic face animation. FIG. 10 shows an example process of thelip synchronization and facial reenactment 325. FIG. 10 shows an image1005 of a facesync actor face, an image 1010 of the facesync actorfacial landmarks, an image 1015 of user facial landmarks, and an image1020 of the user's face with the facial expression of the facesyncactor, according to an example embodiment. The steps of lipsynchronization and facial reenactment 325 may include recording afacesync actor and pre-processing a source video/image to obtain theimage 1005 of a facesync actor face. Then, the facial landmarks may beextracted as shown by the image 1010 of the facesync actor faciallandmarks. This step also may include gaze tracking of the facesyncactor. In some embodiments, instead of recording a facesync actor,previously prepared animated 2D or 3D face and mouth region models maybe used. The animated 2D or 3D face and mouth region models may begenerated by machine learning techniques.

Optionally, fine tuning of the facial landmarks may be performed. Insome example embodiments, the fine tuning of the facial landmarks isperformed manually. These steps can be performed in a cloud whenpreparing the video configuration file. In some example embodiments,these steps may be performed during the resource preparation 315. Then,the user's facial landmarks may be extracted as shown by the image 1015of the user's facial landmarks. The next step of the synchronization andfacial reenactment 325 may include animation of the target image withextracted landmarks to obtain the image 1020 of the user's face with thefacial expression of the facesync actor. This step may be performed on acomputing device based on an image of a face of the user. The method ofanimation is described in detail in U.S. patent application Ser. No.16/251,472, the disclosure of which is incorporated herein by referencein its entirety. The lip synchronization and facial reenactment 325 canalso be enriched with Artificial Intelligence-made head turns.

In some example embodiments, after the user takes an image, a 3D modelof the user's head may be created. In this embodiment, the step of lipsynchronization and facial reenactment 325 may be omitted.

The hair animation 330 may be performed to animate hair of the user. Forexample, if the user has hair, the hair may be animated when the usermoves or rotates his head. The hair animation 330 is shown in FIG. 11.FIG. 11 shows a segmented face image 1105, a hair mask 1110, a hair maskwarped to the face image 1115, and the hair mask applied to the faceimage 1120, according to an example embodiment. The hair animation 330may include one or more of the following steps: classifying the hairtype, modifying appearance of the hair, modifying a hair style, makingthe hair longer, changing the color of the hair, cutting and animatingthe hair, and so forth. As shown in FIG. 11, a face image in the form ofa segmented face image 1105 may be obtained. Then, a hair mask 1110 maybe applied to the segmented face image 1105. The image 1115 shows thehair mask 1110 warped to the face image. The image 1120 shows the hairmask 1110 applied to the face image. The hair animation 330 is describedin detail in the U.S. patent application Ser. No. 16/551,756, thedisclosure of which is incorporated herein by reference in its entirety.

The eyes animation 335 may allow making the facial expressions of theuser more realistic. The eyes animation 335 is shown in detail in FIG.12. The process of eyes animation 335 may consist of the followingsteps: reconstruction of an eye region of the user face, gaze movementstep, and eye blinking step. During the reconstruction of the eyeregion, the eye region is segmented into parts: eyeball, iris, pupil,eyelashes, and eyelid. If some part of the eye region (e.g., iris oreyelid) is not fully visible, the full texture of this part may besynthesized. In some embodiments, a 3D morphable model of an eye may befitted, and a 3D-shape of the eye may be obtained together with thetexture of the eye. FIG. 12 shows an original image 1205 of an eye, animage 1210 with the reconstructed sclera of the eye, and an image 1215with the reconstructed iris.

The gaze movement step includes tracking a gaze direction and pupilposition in a video of a facesync actor. This data may be manuallyedited if the eye movements of the facesync actor are not rich enough.Gaze movements may then be transferred to the eye region of the user bysynthesizing a new eye image with transformed eye shape and the sameposition of iris as that of the facesync actor. FIG. 12 shows an image1220 with the reconstructed moved iris.

During the eye blinking step, the visible part of the eye of the usermay be determined by tracking the eyes of the facesync actor. A changedappearance of eyelids and eyelashes may be generated based on thereconstruction of eye region.

The steps of the eyes animation 335 may be done either explicitly (asdescribed) or implicitly if face reenactment is done using generativeadversarial networks (GAN). In the latter case, the neural network mayimplicitly capture all the necessary information from the image of theuser face and the source video.

During the deploy 340, the user face may be photorealistically animatedand automatically inserted in footage templates. The files from theprevious steps (resources preparation 315, skin recoloring 320, lipsynchronization and facial reenactment 325, hair animation 330, and eyesanimation 335) may be used as data for a configuration file. Examples ofpersonalized videos with a predefined set of user faces can be generatedfor initial review. After the issues that were identified during thereview are eliminated, the personalized video may be deployed.

The configuration file may also include a component that allows toindicate the text parameters for customized personalized videos. Acustomized personalized video is a type of a personalized video thatallows users to add any text the users want on top of the final video.The generating of personalized videos with customized text messages isdescribed in more detail in U.S. patent application Ser. No.______/______, dated ______, titled “SYSTEM AND METHODS FOR GENERATINGPERSONALIZED VIDEOS WITH CUSTOMIZED TEXT MESSAGES,” the disclosure ofwhich is incorporated herein in its entirety.

In an example embodiment, the generation of the personalized videos mayfurther include the steps of generating significant head turns of auser's head; body animation and changing clothes; facial augmentationssuch as hairstyle changing, beautification, adding accessories, and thelike; changing the scene illumination; synthesizing the voice that mayread/sing the text that user has typed or changing the voice over toneto match the voice of the user; gender switching; construction of abackground and a foreground depending on the user input; and so forth.

FIG. 4 is a schematic showing functionality 400 of the system 250 fortemplate-based generation of the personalized videos, according to someexample embodiments. The system 250 may receive an image of a sourceface shown as a user face image 405 and a video template including videoconfiguration data 410. The video configuration data 410 may includedata sequences 420. For example, the video configuration data 410 mayinclude a sequence of frame images, a sequence of face area parametersdefining positions of a face area in the frame images, and a sequence offacial landmark parameters defining positions of facial landmarks in theframe images. Each of the facial landmark parameters may correspond to afacial expression. The sequence of frame images may be generated basedon an animation video or based on a live action video. The sequence offacial landmark parameters may be generated based on a live action videofeaturing a face of a facesync actor. The video configuration data 410may further include a skin mask, eyes parameters, a mouth region image,head parameters, animated object images, preset text parameters, and soforth. The video configuration data may include a sequence of skin masksdefining a skin area of a body of at least one actor featured in theframe images. In an example embodiment, the video configuration data 410may further include a sequence of mouth region images. Each of the mouthregion images may correspond to at least one of the frame images. In afurther example embodiment, the video configuration data 410 may includea sequence of eye parameters defining positions of an iris in a scleraof a facesync actor featured in the frame images and/or a sequence ofhead parameters defining a rotation, a turn, a scale, and otherparameters of a head. In another example embodiment, the videoconfiguration data 410 may further include a sequence of animated objectimages. Each of the animated object images may correspond to at leastone of the frame images. The video configuration data 410 may furtherinclude a soundtrack 450.

The system 250 may determine, based on the user face image 405, userdata 435. The user data may include user facial landmarks, a user facemask, user color data, a user hair mask, and so forth.

The system 250 may generate, based on the user data 435 and the datasequences 420, frames 445 of an output video shown as a personalizedvideo 440. The system 250 may further add the soundtrack to thepersonalized video 440. The personalized video 440 may be generated bymodifying a frame image of the sequence of frame images. The modifyingof the frame image may include modifying the user face image 405 toobtain a further image featuring the source face adopting a facialexpression corresponding to the facial landmark parameters. Themodification may be performed based on facial landmark parameterscorresponding to the frame image. The further image may be inserted intothe frame image at a position determined by face area parameterscorresponding to the frame image. In an example embodiment, thegeneration of the output video may further include determining colordata associated with the source face and, based on the color data,recoloring the skin area in the frame image. Additionally, thegeneration of the output video may include inserting, into the frameimage, a mouth region corresponding to the frame image. Further steps ofthe generation of the output video may include generating an image of aneyes region based on the eye parameters corresponding to the frame andinserting the image of the eyes region in the frame image. In an exampleembodiment, the generation of the output video may further includedetermining a hair mask based on the source face image, generating ahair image based on the hair mask and head parameters corresponding tothe frame image, and inserting the hair image into the frame image.Additionally, the generation of the output video may include inserting,into the frame image, an animated object image corresponding to theframe image.

FIGS. 13-14 show frames of example personalized videos generated basedon video templates, according to some example embodiments. FIG. 13 showsa filmed personalized video 1305 with an actor, in which the recoloringwas performed. FIG. 13 further shows a personalized video 1310 createdbased on a stock video obtained from a third party. In the personalizedvideo 1310, a user face 1320 is inserted into the stock video. FIG. 13further shows a personalized video 1315, which is a 2D animation with auser head 1325 added on top of the 2D animation.

FIG. 14 shows a personalized video 1405, which is a 3D animation with auser face 1415 inserted into the 3D animation. FIG. 14 further shows apersonalized video 1410 with effects, animated elements 1420, and,optionally, text added on top of the image of the user face.

FIG. 15 is a flow chart showing a method 1500 for template-basedgeneration of personalized videos, according to some example embodimentsof the disclosure. The method 1500 can be performed by the computingdevice 105. The method 1500 may commence with receiving videoconfiguration data at step 1505. The video configuration data mayinclude a sequence of frame images, a sequence of face area parametersdefining positions of a face area in the frame images, and a sequence offacial landmark parameters defining positions of facial landmarks in theframe images. Each of the facial landmark parameters may correspond to afacial expression. In an example embodiment, the sequence of frameimages may be generated based on an animation video or based on a liveaction video. The sequence of facial landmark parameters may begenerated based on a live action video featuring a face of a facesyncactor. The video configuration data may include one or more of thefollowing: a sequence of skin masks defining a skin area of a body of atleast one actor featured in the frame images, a sequence of mouth regionimages where each of the mouth region images corresponds to at least oneof the frame images, a sequence of eye parameters defining positions ofan iris in a sclera of a facesync actor featured in the frame images, asequence of head parameters defining a rotation, a scale, a turn, andother parameters of a head, a sequence of animated object images,wherein each of the animated object images corresponds to at least oneof the frame images, and so forth.

The method 1500 may continue with receiving an image of a source face atstep 1510. The method 1500 may further include generating an outputvideo at step 1515. Specifically, the generation of the output video mayinclude modifying a frame image of the sequence of frame images. Theframe image may be modified by modifying the image of the source face toobtain a further image featuring the source face adopting a facialexpression corresponding to the facial landmark parameters. The image ofthe source face may be modified based on facial landmark parameterscorresponding to the frame image. The further image may be inserted intothe frame image at a position determined by face area parameterscorresponding to the frame image. In an example embodiment, thegeneration of the output video may further optionally include one ormore of the following steps: determining color data associated with thesource face and recoloring the skin area in the frame image based on thecolor data, inserting a mouth region corresponding to the frame imageinto the frame image, generating an image of eyes region based on theeye parameters corresponding to the frame, inserting the image of theeyes region in the frame image, determining a hair mask based on thesource face image and generating a hair image based on the hair mask andhead parameters corresponding to the frame image, inserting the hairimage into the frame image, and inserting an animated object imagecorresponding to the frame image into the frame image.

FIG. 16 illustrates an example computing system 1600 that can be used toimplement methods described herein. The computing system 1600 can beimplemented in the contexts of the likes of computing devices 105 and110, the messenger services system 130, the messenger 220, and thesystem 250 for template-based generation of personalized videos.

As shown in FIG. 16, the hardware components of the computing system1600 may include one or more processors 1610 and memory 1620. Memory1620 stores, in part, instructions and data for execution by processor1610. Memory 1620 can store the executable code when the system 1600 isin operation. The system 1600 may further include an optional massstorage device 1630, optional portable storage medium drive(s) 1640, oneor more optional output devices 1650, one or more optional input devices1660, an optional network interface 1670, and one or more optionalperipheral devices 1680. The computing system 1600 can also include oneor more software components 1695 (e.g., ones that can implement themethod for template-based generation of personalized videos as describedherein).

The components shown in FIG. 16 are depicted as being connected via asingle bus 1690. The components may be connected through one or moredata transport means or data network. The processor 1610 and memory 1620may be connected via a local microprocessor bus, and the mass storagedevice 1630, peripheral device(s) 1680, portable storage device 1640,and network interface 1670 may be connected via one or more input/output(I/O) buses.

The mass storage device 1630, which may be implemented with a magneticdisk drive, solid-state disk drive, or an optical disk drive, is anon-volatile storage device for storing data and instructions for use bythe processor 1610. Mass storage device 1630 can store the systemsoftware (e.g., software components 1695) for implementing embodimentsdescribed herein.

Portable storage medium drive(s) 1640 operates in conjunction with aportable non-volatile storage medium, such as a compact disk (CD), ordigital video disc (DVD), to input and output data and code to and fromthe computing system 1600. The system software (e.g., softwarecomponents 1695) for implementing embodiments described herein may bestored on such a portable medium and input to the computing system 1600via the portable storage medium drive(s) 1640.

The optional input devices 1660 provide a portion of a user interface.The input devices 1660 may include an alphanumeric keypad, such as akeyboard, for inputting alphanumeric and other information, or apointing device, such as a mouse, a trackball, a stylus, or cursordirection keys. The input devices 1660 can also include a camera orscanner. Additionally, the system 1600 as shown in FIG. 16 includesoptional output devices 1650. Suitable output devices include speakers,printers, network interfaces, and monitors.

The network interface 1670 can be utilized to communicate with externaldevices, external computing devices, servers, and networked systems viaone or more communications networks such as one or more wired, wireless,or optical networks including, for example, the Internet, intranet, LAN,WAN, cellular phone networks, Bluetooth radio, and an IEEE 802.11-basedradio frequency network, among others. The network interface 1670 may bea network interface card, such as an Ethernet card, optical transceiver,radio frequency transceiver, or any other type of device that can sendand receive information. The optional peripherals 1680 may include anytype of computer support device to add additional functionality to thecomputer system.

The components contained in the computing system 1600 are intended torepresent a broad category of computer components. Thus, the computingsystem 1600 can be a server, personal computer, hand-held computingdevice, telephone, mobile computing device, workstation, minicomputer,mainframe computer, network node, or any other computing device. Thecomputing system 1600 can also include different bus configurations,networked platforms, multi-processor platforms, and so forth. Variousoperating systems (OS) can be used including UNIX, Linux, Windows,Macintosh OS, Palm OS, and other suitable operating systems.

Some of the above-described functions may be composed of instructionsthat are stored on storage media (e.g., computer-readable medium orprocessor-readable medium). The instructions may be retrieved andexecuted by the processor. Some examples of storage media are memorydevices, tapes, disks, and the like. The instructions are operationalwhen executed by the processor to direct the processor to operate inaccord with the invention. Those skilled in the art are familiar withinstructions, processor(s), and storage media.

It is noteworthy that any hardware platform suitable for performing theprocessing described herein is suitable for use with the invention. Theterms “computer-readable storage medium” and “computer-readable storagemedia” as used herein refer to any medium or media that participate inproviding instructions to a processor for execution. Such media can takemany forms, including, but not limited to, non-volatile media, volatilemedia, and transmission media. Non-volatile media include, for example,optical or magnetic disks, such as a fixed disk. Volatile media includedynamic memory, such as system random access memory (RAM). Transmissionmedia include coaxial cables, copper wire, and fiber optics, amongothers, including the wires that include one embodiment of a bus.Transmission media can also take the form of acoustic or light waves,such as those generated during radio frequency (RF) and infrared (IR)data communications. Common forms of computer-readable media include,for example, a floppy disk, a flexible disk, a hard disk, magnetic tape,any other magnetic medium, a CD-read-only memory (ROM) disk, DVD, anyother optical medium, any other physical medium with patterns of marksor holes, a RAM, a PROM, an EPROM, an EEPROM, any other memory chip orcartridge, a carrier wave, or any other medium from which a computer canread.

Various forms of computer-readable media may be involved in carrying oneor more sequences of one or more instructions to a processor forexecution. A bus carries the data to system RAM, from which a processorretrieves and executes the instructions. The instructions received bythe system processor can optionally be stored on a fixed disk eitherbefore or after execution by a processor.

Thus, the methods and systems for template-based generation ofpersonalized videos have been described. Although embodiments have beendescribed with reference to specific example embodiments, it will beevident that various modifications and changes can be made to theseexample embodiments without departing from the broader spirit and scopeof the present application. Accordingly, the specification and drawingsare to be regarded in an illustrative rather than a restrictive sense.

What is claimed is:
 1. A method for template-based generation ofpersonalized videos, the method comprising: receiving, by a computingdevice, video configuration data including: a sequence of frame images;a sequence of face area parameters defining positions of a face area inthe frame images; and a sequence of facial landmark parameters definingpositions of facial landmarks in the frame images, wherein each of thefacial landmark parameters corresponds to a facial expression;receiving, by the computer device, an image of a source face; andgenerating, by the computing device, an output video, wherein thegenerating the output video includes modifying a frame image of thesequence of frame images by: modifying, based on facial landmarkparameters corresponding to the frame image, the image of the sourceface to obtain a further image featuring the source face adopting afacial expression corresponding to the facial landmark parameters; andinserting the further image into the frame image at a positiondetermined by face area parameters corresponding to the frame image. 2.The method of claim 1, wherein the sequence of frame images is generatedbased on one of the following: an animation video and a live actionvideo.
 3. The method of claim 1, wherein the sequence of facial landmarkparameters is generated based on a live action video featuring a face ofa face synchronization actor.
 4. The method of claim 1, wherein: thevideo configuration data include a sequence of skin masks defining askin area of a body of at least one actor featured in the frame imagesor a skin area of 2D/3D animation of a further body; and the generatingthe output video includes: determining color data associated with thesource face; and recoloring, based on the color data, the skin area inthe frame image.
 5. The method of claim 1, wherein: the videoconfiguration data further include a sequence of mouth region images,each of the mouth region images corresponding to at least one of theframe images; and the generating the output video includes inserting,into the frame image, a mouth region corresponding to the frame image.6. The method of claim 1, wherein: the video configuration data furtherinclude a sequence of eye parameters defining positions of an iris in asclera of a face synchronization actor featured in the frame images; andthe generating the output video includes: generating, based on the eyeparameters corresponding to the frame, an image of eyes region; andinserting the image of eyes region in the frame image.
 7. The method ofclaim 1, wherein: the video configuration data include a sequence ofhead parameters defining one or more of a rotation, a turn, a position,and a scale of a head.
 8. The method of claim 1, wherein: the generatingthe output video includes: determining, based on the image of the sourceface, a hair mask; generating, based on the hair mask, a hair image; andinserting the hair image into the frame image.
 9. The method of claim 1,wherein: the video configuration data include a sequence of animatedobject images, wherein each of the animated object images corresponds toat least one of the frame images; and the generating the output videoincludes inserting, into the frame image, an animated object imagecorresponding to the frame image.
 10. The method of claim 1, wherein:the video configuration data include a soundtrack; and the generatingthe output video further includes adding the soundtrack to the outputvideo.
 11. A system for template-based generation of personalizedvideos, the system comprising at least one processor and a memorystoring processor-executable codes, wherein the at least one processoris configured to implement the following operations upon executing theprocessor-executable codes: receiving, by a computing device, videoconfiguration data including: a sequence of frame images; a sequence offace area parameters defining positions of a face area in the frameimages; and a sequence of facial landmark parameters defining positionsof facial landmarks in the frame images, wherein each of the faciallandmark parameters corresponds to a facial expression; receiving, bythe computer device, an image of a source face; and generating, by thecomputing device, an output video, wherein the generating the outputvideo includes modifying a frame image of the sequence of frame imagesby: modifying, based on facial landmark parameters corresponding to theframe image, the image of the source face to obtain a further imagefeaturing the source face adopting a facial expression corresponding tothe facial landmark parameters; and inserting the further image into theframe image at a position determined by face area parameterscorresponding to the frame image.
 12. The system of claim 11, whereinthe sequence of frame images is generated based on one of the following:an animation video and a live action video.
 13. The system of claim 11,wherein the sequence of facial landmark parameters is generated based ona live action video featuring a face of a face synchronization actor.14. The system of claim 11, wherein: the video configuration datainclude a sequence of skin masks defining a skin area of a body of atleast one actor featured in the frame images or a skin area of 2D/3Danimation of a further body; and the generating the output videoincludes: determining color data associated with the source face; andrecoloring, based on the color data, the skin area in the frame image.15. The system of claim 11, wherein: the video configuration datafurther include a sequence of mouth region images, each of the mouthregion images corresponding to at least one of the frame images; and thegenerating the output video includes inserting, into the frame image, amouth region corresponding to the frame image.
 16. The system of claim11, wherein: the video configuration data further include a sequence ofeye parameters defining positions of an iris in a sclera of a facesynchronization actor featured in the frame images; and the generatingthe output video includes: generating, based on the eye parameterscorresponding to the frame, an image of eyes region; and inserting theimage of eyes region in the frame image.
 17. The system of claim 11,wherein: the video configuration data include a sequence of headparameters defining one or more of a rotation, a turn, a position, and ascale of a head.
 18. The system of claim 11, wherein: the generating theoutput video includes: determining, based on the source face image, ahair mask; generating, based on the hair mask, a hair image; andinserting the hair image into the frame image.
 19. The system of claim11, wherein: the video configuration data include a sequence of animatedobject images, wherein each of the animated object images corresponds toat least one of the frame images; and the generating the output videoincludes inserting, into the frame image, an animated object imagecorresponding to the frame image.
 20. A non-transitoryprocessor-readable medium having instructions stored thereon, which whenexecuted by one or more processors, cause the one or more processors toimplement a method for template-based generation of personalized videos,the method comprising: receiving, by a computing device, videoconfiguration data including: a sequence of frame images; a sequence offace area parameters defining positions of a face area in the frameimages; and a sequence of facial landmark parameters defining positionsof facial landmarks in the frame images, wherein each of the faciallandmark parameters corresponds to a facial expression; receiving, bythe computer device, an image of a source face; and generating, by thecomputing device, an output video, wherein the generating the outputvideo includes modifying a frame image of the sequence of frame imagesby: modifying, based on facial landmark parameters corresponding to theframe image, the image of the source face to obtain a further imagefeaturing the source face adopting a facial expression corresponding tothe facial landmark parameters; and inserting the further image into theframe image at a position determined by face area parameterscorresponding to the frame image.